High-order Deep Neural Networks for Learning Multi-Modal Representations

In multi-modal learning, data consists of multiple modalities, which need to be represented jointly to capture the real-world ’concept’ that the data corresponds to (Srivastava & Salakhutdinov, 2012). However, it is not easy to obtain the joint representations reflecting the structure of multi-modal data with machine learning algorithms, especially with conventional neural networks. This is because the information which consists of multiple modalities has distinct statistical properties and each modality has a different kind of representation and correlational structure (Srivastava & Salakhutdinov, 2012). Also, noise exists in information from multi-modal input, which makes the information unreliable and inaccurate (Ernst & Di Luca, 2011).